For many operations, a human operator manipulates an input device in an environment local to the operator, and the effect of the manipulation takes place in a remote environment, by virtue of the actions of an effector located in the remote environment. Such operations include the manipulation of effectors: in outer space by ground based operators; in hazardous environments, such as a nuclear reactor core, by operators a safe distance away; attached to construction equipment, such as cranes by operators distant from the crane. To facilitate the control of the effector, various forms of feedback have been used. Visual feedback, typically using a video camera and visual display device is very helpful. However, in many situations, visual feedback is not available, such as when the remote environment is dark or murky, for instance below water. Visual feedback can be degraded due to the various complexities involved in video transmission. Further, video transmission requires a rather large bandwidth and relatively expensive equipment. Also, due to the arrangement of the effector with respect to the remote environment, the view of the visual capture device, such as a camera, is often blocked by intervening elements.
It is also common to provide a physically reactive force feedback. In such an arrangement, the effector is provided with an array of force sensors, which are designed and located to measure the forces applied to the effector by interaction with its environment. These forces are transduced and transmitted to the environment of the local operator, where they are retransduced by motor and gear systems into forces that are applied to the input apparatus that the operator uses. Thus, the operator "feels" the forces that the effector feels, to some degree depending on the fidelity of the force sensing and reproduction scheme.
Force information is useful for several reasons. It facilitates governing the forces applied by the effector, either to avoid excessive stresses or to produce a desired movement of an object, or to minimize energy expenditure. It also facilitates estimating mass, frictional resistance, or other properties of objects in the remote environment in order to predict their behavior. It also provides information about contact with objects or with features of the environment, as in searching when visual information is unavailable. In some cases, it also guides the operator's body (typically the hand) to an appropriate location.
Use of reactive force feedback has drawbacks. The apparatus required to reproduce the force at the local environment of the operator may be costly, complicated and bulky. Further, the force fed back to the operator's hand acts as an additional input to the input device, thereby having an effect on the remote effector. It is also often desirable to amplify the force feedback signal, to enable sensing modest or delicate forces. If the gain on the feedback signal is large, in order to provide needed sensitivity, a small force at a remote location would be presented as a large force to the operator. This would impose a large force on the operator and could make the input controller more difficult to move. Thus the task would be made more physically difficult, the operation would be slowed down, and the rate of operator fatigue would be increased. If gains are made excessively high, instability could arise.
Some force feedback systems present the feedback with a time delay. Time delays on the order of several seconds are common in connection with communication to outer space, even with the signal being transmitted at the speed of radio waves. It has generally been found that reactive feedback in the presence of time delays can not be used effectively because instabilities quickly arise. See generally, W. R. Ferrel, "Delayed Force Feedback,: Human Factors, October 1986, pp. 449-455.
It has been noted that if reactive force feedback is provided to the operator, not to the hand that is applying the force, but to the other hand, the operator gains an advantage. See S. Weissenberger and T. B. Sheridan, "Dynamics of human operator control systems using tactile feedback," A.S.M.E. Transactions (J. Basic Eng.), vol. 84, pp. 297-301 (1962). However, the fact that the feedback is provided to another limb, and, in fact, an oppositely disposed limb, leaves room for improvement.
It has also been attempted to provide force feedback through the visual modality, such as by using a visual display that indicates force, either through numbers, or a pictorial representation. However, the visual modality is typically heavily burdened by other demands. It has been found that such methods typically overload the operator's ability to process visually perceived information, thus leading to degraded or unacceptable performance.
There are other situations where feedback of force is desirable, which are not obviously within the class of remote effector applications. For instance, in microsurgery, or any micromanipulation, the operator manipulates an input device with a resultant motion by an effector, perhaps very close to the operator, yet connected by a linkage (mechanical, electrical, electromechanical, or other) that effectively distances the operator from the resultant activity as if the operator were far away. Such microsurgery could be conducted over distances of either inches or miles, with the appropriate visual feedback tools. Another instance relates to the control of artificial limbs, for instance an artificial arm or hand. One type of artificial limb is referred to as myeoelectric. Such a device is connected to the receive impulses from the nerves of the muscles remaining in the stump of the limb. The patient still has the ability to move these muscles, thereby generating electric signals, which can be used as inputs to the prosthesis. It would also be helpful if the patient were aware of the force field in which the prosthesis fingers, hand, wrist, elbow, etc. operates. Force related cues could provide valuable information to the patient. However, reactive force feedback is not practical in such a situation because the user has no sensation in the artificial limb.
Previous work of the present inventor has indicated that a vibrotactile display can be used to indicate whether or not a remote effector has come into contact with an object in the remote environment. In that work, a sensor/transducer carried by an effector was connected to a computer, which generated a signal to a vibrating unit fixed to the hand of the user that guided the input device. The computer generated signal was not related in any way to the magnitude of the force experienced by the sensor, other than that the signal indicated a positive force if the force exceeded the minimal sensitivity threshold of the sensor. No information was provided to the user about the magnitude or the direction of the force. See N. J. M. Patrick "Design, Construction, and Testing of a Fingertip Tactile Display for Interaction with Virtual and Remote Environments," submitted in partial fulfillment of the requirements for the degree of Master of Science in Mechanical Engineering at the Massachusetts Institute of Technology pp. 2 and 63-67 (1990), citing the present inventor's unpublished research. The entire document is hereby incorporated fully herein by reference. The principal focus of the Patrick work was using a vibrotactile display to track the location of the user's fingers relative to desired locations, particularly in the context of a virtual reality glove manipulation. The signal generated by the display device was a vibration having an amplitude linearly related to the distance between the location of the fingers and the desired location.